Skip to content

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 22, 2025

What changes were proposed in this pull request?

This PR aims to support TLS connections between Spark Connect Client for Swift and Spark Connect Server.

Why are the changes needed?

For feature parity.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-51529] Support for TLS connections [SPARK-51529] Support TLS connections Apr 22, 2025
@grundprinzip
Copy link

While experimenting with the client, I found another issue: It will always create a new connection and not reuse an existing connection for the session.

Every time where withGRPC is called (https://github.com/apache/spark-connect-swift/blob/main/Sources/SparkConnect/SparkConnectClient.swift#L96) it will instantiate a new connection.

It would be great to make the connection stateful because otherwise it will be a lot of overhead of establishing connections and every time doing a TLS handshake.

But this can easily be addressed in a follow up PR.

@dongjoon-hyun
Copy link
Member Author

Thank you for verifying TLS feature, @grundprinzip .

Yes, reusing and reconnection feature is related to many implementation details. So, I'm stilling working on it.

Merged to main!

@dongjoon-hyun dongjoon-hyun deleted the SPARK-51529 branch April 22, 2025 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants